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ABSTRACT 

The present paper sought to investigate the 
usefulness to accountability of identifying outlying schools through 
multiple regression. The outlier approach identifies schools vhich 
differ significantly for reasons other than variation in the 
predictors. The present study was not able to confirm the construct 
validity of the outlier approach for the author's school system* That 
is, educational process variables (observations, interviews, school 

size, staff ratio s, abil it y sco re s, etc. ) d id no t discriminate the 

posiTive fromnfHST negative outliers. Moreover, ^The study sKoweS^the 
outlier approach may be merely a tautology (i»e*, 5!t of the schools 
are outliers at the S% level of significance •) (Author) 
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OmiEBS ACCOinmBIUSIi PACT OH EICTIOH? 

Hecently^ the use of outliers to identify earoeptionally acbievin^ 
echools b&B been proposed or noted In several acoountability systema (e.g.^ 
KUtgaard^ 1973} Maagrland State Department of Education, 1975). Essen- 
tially, an outlier is a data element extremely distant £xom the central 
tendency of its distribution. **E)ttreni9ly distant'^ is typically defined 
in mxoh the asm way as '^statistical significance*'! often the sasoe per- 
centiles are used as cutoff points. Itetection of outliers in industrial 
gusli^ control is commonplace (e.g., Zwidcl, 1975); detection of outlying 
schools in educational acoountability is eioerging (Cooperative Account- 
ability Project, 1975). 

In the Maryland Accountability Program (HAP), outliers axe located 
iBT ^tBe^foIIoiSng" manner . Scdiool mean achievement scores are regi^ssed 
upon a set of predictors (e.g., school mean ability scores, median femily 
income). Sesiduals axe calculated for each school. Any school ^se 
absolute residual is larger than a given value is identified as an outlier. 

Appxoxijnately ^1.96 standard deviations £i:om the mean of the reside 
ual distrlbuti^ (zero) axe used as the cutoff points, yielding the 
2^ and 97^ percentile points assuming noimlity, respectively. 

Supposedly, a school in the upper tail of the residual distribution 
(termed ''positive outliers'*) is exhibiting average achievement signifi- 
cantly greater than e:i9eoted (or predicted), based upon its average 
ability, socioeconomic levels or other vaxiables# Inversely, a sdiocl 
labeled as a *^gative outlier** has significantly lower average achievement 
than expected. 
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The regression process voi^ much aa a blocking prooedure^ idierein 
schools are grouped }>y abilities (or eocioeconomi^i levels). Ertreme 
schooler if any ezxst wltiii^ a blocks are -Oien identified as outliers* 

Once schooXa are denoted aa outliers » the nesct step ^ ^ }&u:ylaiui 
Accountability Flan is a process analyais of observable educational var^ 
iables for attribution of -Qie test* results. !I!!hoxig}i -Qiere are several 
statistical problems with -Oie use of regression analysis (e*g*r correlated 
predictors (Luecke and McGinn^ 1975)^ reliability of the data (Stanley^ 
1971) the present paper is concerned only wi-Qi a logical analysis of 
regression and outliers in accountability. 

Xt is the au-Qior^s contention that the use of regression to identiiy 
outliers may constitute a tautology^ not related at all to process var^ 

iables of educational ^i^iificance^j)3r^to_i^^ 

(PXeaae note that this paper is not a proven theorem; it consists of 
thou£^ts and concerns which hopefully will provoke reflection and con- 
sideration about accountability assessment). 

Let us e^tsmine an esanifple of test score data £rom ^ t!!itle X 
class. Seferring to tSable 1^ ve bave^ for twenty*-five pupils^ their 
reading score (the predictor)^ math score (the criterion)^ predicted 
ma-Qi score (the point on the regressicm line (X^t) and the residual 

A 

(-Qie algebraic difference between T and T. Some coraon facts about 
regression lines are apparent £rcm the table: the mean of -Qie regression 
residuals is sero^ since T « E(T|X); Hib mean of the predicted math scores 
is eqpBl to the mean of -Qie actu^ math scores; the standard deviation 
of -Qie predicted math scores is smaller than the standard deviation 
of -Qie actual math scores^ since soma of the error variance has been 
reduced 1:^ predictijagt the standard deviation of the residuals is the 
vell'knovn standard error of estimate. 
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Accoxains to the K^xyland Accountability Progxsjn^ aiqr data element 
(in tMd case» pupils), ^dioee absolute reeidual is larger than 1-96 
standard exrore of estimate ie an outlier. Pupil 23 is the outlier 
in Idiid exac^le. 

Fiffrce 1 giaphs the (X»Y) data pointe and the rsgxedsicm line 
Figure 2 ehovs a hietogram ot the reeiduals. Evan for 
only 25 cases, the t^ntogsm is ramaiil^ably mound^shaped. 

Nowr vjhere does the claim of a tautology surface? If our cutoff 
pointe encoo^e five percent of the residual distri'mticmr ve wcmld 
es^ot to identiiy five percent of the data elements as outXiers. 
In the above exan^le, we would expect one outXierj wa obtained one 
outlier (pupil 23). 

_ The^^ltimore_6g5mty 
the lyik accountability aaseeainsnt, 6 vara outliers in reading (we 
would e^^ot 5-U); i^ 1975» $ sdiools were outliers (two echools vers 
outliers both years). 

Hotioe also, in the exaniple, that 4^.0 etandaxd errors of estimate 
f^^om sero ^aolose approximately tvo*tMrds of the reeidualsr a result 
entirely predicted ftom normal curve theory. 

Ihus it seems plauaible^ tl»it identii^^ing outlying residuals teem 
the data upon i&ioh the rseidoals are calculated 0^ be no mors "^lan a 
classical ^t^ypo I error. Ie it the case that locating a school in the 
tails of the residual distribution is neither neoeseaiiy nor sufficient 
for observing educationally elgiiifioant processee? (See also I^mcan^ 

^ther difficulty in e^laining the use of outliers i^ account* 
ability is that a positive outlier school may have lower average achiavo* 
fflsnt^ than a negative outlier echool. Suppose, we. enoounter„a..eohcol. .tj^iose 
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avarage ability Is 110^ and averago acMeveisent in the fiJTth grade is 
5*3- Mo-Qier sobool bas an avexaga ability of 9$ and averd^ achievement 
of h*$* ^ShB tLrat school has significantly low achievement for Its 
ability^ the second school has significantly hi^ aohievamdnt for its 
ability. Ebvever^ wcxtld observations of the processes in the second 
school be e:q?eoted to help the first school? How do ve account for 
error in the tested ability? How do ve transfer processes betvson two 
schools so unalilce? 

Certainly: If we had two schools of ecfual ability^ one of which 
^s a positive outlier and the other a negative outlier^ ve jni^t e^ect 
to profit 1:^ a process evaluation. Btit the present regression method 
does not necessarily identiiy such pairs* 

It vould seem that the process evaluati on iQ necessarj^tojralidat^ 

the statistical identifioation procedures* rather than to eicplain the 
results* 

Mthough the Naiyland Aooountabillty Program process evaluation has 
not yet ooourxed* -Qie present author has conducted throe such similar 
evaluations on a mach smaller scale in the Saltimosre County sohoo7s* 
III each oase* the process evaluation failed to oonfixm the construct 
validity of the statistical identification. Of course* it seems 
imperative that the process evaluators not know vdiioh schools categorized 
as positive or negative outliers* or the results ^^re obvious (this 
*%lind" evaluation appears close to Sorivea's goal-£^e evaluation). 

Ill the first study* instructional supervisors observed schools 
some of ^ch had recorded outlying gains in aoMevement (positive and 
negative)* some of tdixch had not. SThe data vsre collected using nan^a^ 
tives and interviews. No differences among -Oie positive and negative 
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outliers vare obaervod. The aecond &tu4y ijavolved Title I sohoola 
^^oh were otitliexs. Project ataff observed in these achoola but vers 
unable to diacem vdiich achoola the outliers. TbB thlx^d study 
al£?o iBToXved Title I achools^ but the observers wore BOt on th& 
county staff and sane used fouoal obaorvational acalea. Once again^ 
the aearoh for procesaea was not fmiitfUl. 

In addition^ a glance at aeven baclcgzaund varxablea for the three 
poaitive and three negative outliers identified in the 197U M^land 
accountability teatin^ reveals no differences in their avera^ enroll- 
montSy average abilitiea^ pupil/ataff^ average years ataff eiiporiencey 
percent of etaff with advanced degrees^ percent disadvantogedt and 
jnedian Ibmily incomo (1970 dollars). If there are other educational 
variables ex^ain th ree a chools' ^hiev^enta^ are aignif^^ 

cantly higher than their abilitiee predict and three are lovert 
they are not encumbered with the school resource or input variables 
usually encountered. 

Of coTirsef lack of power in the process evaluation to detect the 
causes of tbe statistical differences is a plausible explenatic^at 
but eo is a !!!^e I error. That ist the outli&rs are part of the 
error variance of tbB sregrcssion analysis and not of ai^ educatioiial 
significance. 

Several future courses may be worth^diile to pursue. SThe present 
au'tiior welcomes conimentsy criticisms and suggestions. 

(1) Consider schools for process evaluation only if Miey are 
outliers at least two years in a row. (e.g.t Klitgaard^ 
1973) 

(2) Consider schools for process evaluation only if they are 
outliers on most of Mis subtests of Mis test being xised. 
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(3) ^86 a xfitziaQm Bangle of schools not statintical outliexd 
in -Qie process evaluation ea well aa tho^a schools so 
identified as outliexa. 

(U) Txy to develop a theoretical equation agaijist vhich to 
test for outliers (possibly a given year*s xa^ssim 
equatioa-^ori of a ncrmlnf prooeds)^ rather than usiJ3£ 
the data to generate their cm outliers* 

(5) K&ep -Qie process evaluation a ''blind*' e^qperiisent relative 
to the statistical results* 

(6) Don't count the nuntber of outliers cosrpaxed to an 
expected mmiberr unless -Qie distribution of residuals 
can be aseuoed normal (or it'd paxaffietere Imown) 

(7) C(Miaider the use of blocks or groups of schools alike 
in ability (or whatever covariate ie used) rather tiian 
regression aa a tool to locate unusually achieving 
schools* BLocking ia £u: essier to eig^lain to a school 
VTA than is regression analysis* 



Xhfi present paper has discussed a concern about using regression 
analyBis to idMrtjijr outly iiig schools in puisuipg acc ountabilit y 



assessment. 
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